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ABSTRACT 

Recent reports of high levels of achievement on the Regents 
examinations in New York state and relatively poor showing on the eighth 
grade examinations have led many to speculate that the eighth grade 
examinations lack good predictive properties with respect to the Regents 
examinations. There are many problems with arriving at such a conclusion, 
including the unavailability of data that track individual students across 
these two levels of examination and the confounding includence of intervening 
variables even if these individual data were available. This study used 
district-level data from two sources: a sample of 306 districts that 
administered the June 2000 Regents Comprehensive Examination in English (CEE) 
and a special administration of the CEE in April 2000 to seniors who had not 
yet passed the REgents and who missed the January administration because of 
snow. The limitations of these data call for caution about their 
interpretation. These district-level data were matched with the spring 2000 
performance at the district level on the grade 8 English language arts and 
grade 4 English language arts examinations. Results show a good predictive 
relationship among the scores on these examinations. The results of the 
English language arts examinations are grouped into performance levels from 1 
(low) to 4 (high) . Particularly strong was the relationship between the 
representation of the students in level 1 and subsequent performance of the 
students * districts on the Regents CEE. An appendix presents data on agreement 
error. (SLD) 
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Executive Summary 



The New. York State Learning Standards are organized into grade blocks: k - 4, 5 
- 8, and 9-12. The examinations associated with the standards come at the end of these 
blocks. Each examination is designed to certify the level of achievement of the Learning 
Standards, and it follows that two secondary functions are served by the examination 
results: 

1. School, district, and State accountability; 

2. Evaluation of preparedness to achieve at the next higher level. 

Recent reports of high levels of achievement on the Regents examinations and 

relatively poor showing on the eighth grade examinations have led many to speculate that 
the eighth grade examinations lack good predictive properties with respect to the Regents 
examinations. There are many problems with arriving at such a conclusion, including the 
unavailability of data that tracks individual students across these two levels of 
examination, and the confounding influence of intervening variables even if these 
individual data were available. 

Nevertheless, a study is presented using district level data from two sources: a 
Sarriple of 306 districts that administered the June 2000 Regents Comprehensive 
Examination in English (CEE), and a special administration of the Regents CEE in April 
2000 to seniors who had not yet passed the Regents and who missed the January 2000 
administration because of snow. The limitations of these data recommend cautions 
discussed in great detail below about their interpretation. These district level data were 
matched with the spring 2000 performance at the district level on the Grade Eight English 
Language Arts (ELA-8) and Grade Four English Language Arts (ELA-4) examinations. 



Results showed a good predictive relationship among the scores on these examinations. 
The results of the ELA-4 and ELA-8 are grouped into performance levels from 1 (low) to 
4 (high). Particularly strong was the relationship between the representation of the 
students in level 1 and subsequent performance of the students' districts on the Regents 
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Purpose 

This study evaluates the relationship among performance on three State 
examinations of English Language Arts: the Grade Four, Grade Eight, and 
Commencement Level examinations. It is designed to further document construct 
validity of these examinations through predictive sources of evidence. 
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Validity Focus: Sources of Evidence 

In promulgating the New York State Learning Standards in 1995, the State 
undertook the associated responsibility of accountability at the student, school, district, 
and State levels. This responsibility has, as its keystone, development and 
implementation of a new generation of assessments that are sensitive to success and 
growth in the acquisition of the knowledge and skills specified in the Learning Standards 
and their derivative performance indicators. 

The primary property for these instruments to succeed in this responsibility is 
construct validity'. This validity is manifest in a wide variety of evidence, including 
sensitivity to growth across populations^ and evidence of appropriate convergent and 
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discriminant properties^. Predictive evidence can be used to support construct validity by 
documenting that performance on one level, in a prerequisite test, is related to 
performance on tests in the same subject area at more advanced grades. We would 
expect, for example, that results of the Grade Four English Language Arts (ELA-4) 
examination and the Grade Eight English Language Arts (ELA-8) examination evidence 
substantial correlations with the Regents Comprehensive Examination in English (CEE), 
a graduation requirement, and that the ELA-8 results are better related to the ELA-4 
results and to the CEE results than are the ELA-4 results to the CEE results simply 
because the ELA-4 is designed to test prerequisites of ELA-8 and ELA-8 is designed to 
test prerequisites of the CEE more directly. 



Predictive Support of Construct Validity 

The nature of testing in New York State has shifted from documenting 
achievement of coursework to achievement of the Learning Standards. In the former 
environment, demonstrating achievement on the Regents examinations certified 
achievement in courses and general preparedness for graduation. In the current 
environment of Learning Standards, achievement on the new Regents examinations,- 
which are not necessarily course bound, certifies eligibility for graduation and documents 
acquisition of the required standards and performance indicators. While the differences 
may seem subtle, they have enormous implications for the types of evidence and the 
sources of that evidence in support of the validity of the State assessments. 

Before adoption of the Learning Standards, a New York State diploma assured 
that the student had completed the coursework necessary. Inferences about preparedness 
for college or employment were implicit, and depended first on evaluation of the content 
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of the test, and then on inferences about how that content was related to the required 
activities of employment or college work. Since implementation of the Learning 
Standards, the assessments certify that the student has acquired the knowledge and skills 
specified by the learning standards. Very simply, what was an implicit domain has 
become an explicit domain that is given operational definition by the Learning Standards 
and the derivative performance indicators adopted by the Board of Regents. 

While predictive evidence, before, was an important documentation that the Regents 
examinations were sensitive to the skills utilized after graduation, predictive evidence, 
now, is only important insofar as the criterion measures require application of the skills 
and knowledge specified by the Learning Standards. More broadly, the sources of 
evidence most directly related to validity in the new assessment environment are those 
that support that the domain tested is the construct defined by the Learning Standards. 

The purpose of state testing is accountability. This means that the tests must 
certify whether or not the Learning Standards have been acquired. They then enable an 
evaluation of the State, the districts, and the schools, each in their respective roles of 
providing the standards. Sensitivity of the Regents examinations to the Learning 
Standards enables them to certify whether or not the students have acquired this well- 
defined domain of skills and knowledge. Predictive evidence, then, becomes one source 
for documenting this sensitivity and for documenting construct validity of the tests for 
this certifying function, and it is not the sine qua non of validity, as it is for instruments 
designed to tell employers or admission officers of more general readiness to learn or 
perform. 

To illustrate, we would expect that an aptitude instrument that is designed to 
demonstrate preparedness for college, should have a predictive relationship for general 
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performance in the first year of college. On the other hand, with the State Regents 
examinations, the relationships should be well-specified to docmnent validity. Thus, the 
mathematics Learning Standards and performance indicators should first be demonstrated 
as prerequisite to specific courses in colleges, or specific activities of employment after 
high school. Then the relationship of performance on these job activities or in these 
courses to performance on the Regents Mathematics A examination becomes important 
support for the validity of the Mathematics A test. 

Fourth and Eighth Grade Examinations 

Keeping these distinctions in mind, sources of validity evidence of the grade four 
and grade eight instruments can be predictive in nature because, these instruments, as 
well, are designed to certify acquisition of the appropriate Learning Standards for the 
respective grade levels. Without contamination from other variables, performance on 
each of these instruments should be a gauge of performance on the next level of the State 
test in that subject matter. This is because: (1) the tests are each measures of acquisition 
of the Learning Standards, (2) the Learning Standards for each block of grades are 
prerequisite for achievement of the appropriate standards for the next block of grades, 
and (3) the tests for the next block of grades should be sensitive to achievement of the 
respective Learning Standards. 

Limitations on Estimating Relationships 

There are several major obstacles to estimating the relationships among these 
instruments. The first is that New York State does not have an individual student data 
tracking system. Although individual performances are available on the ELA-4, ELA-8, 



Math-4, and Math-8, they are not routinely available for any of the Regents examinations. 
Rather, available data consist of: 

1 . Field test and pretest data; 

2. Survey data estimating proportions of a cohort achieving 
highest scale scores of 0 - 54, 55-64, 65-84, and 85-100 over 
the course of three administrations of the Regents examinations 
per year; 

3. Individual student data generated by the special administration of 
the CEE in April 2000 to seniors who had not yet passed the 
examination. The centralized scoring of this administration 
made these data available, but the restriction of range limited 
their utility; 

4. A sample of several hundred papers drawn from June administrations 
to audit results (Department Review). 

Even though these sources provide some access to student level data, such 
individual data may not be the best source for predictive evidence for the following 
reasons: 

1 . The tests in different grades are administered at different developmental and 
academic stages of the student’s career; 

2. The degree and intensity of academic intervention is in direct relationship 
to performance on the examination. That is, a parity effect exists in which 
the lowest scoring students are provided with the most intense intervention, 
which then has the effect of raising scores on the subsequent State tests; 

3. The ELA-4 and ELA-8 are both census tests administered to the whole 

Of those two classes, while the Regents examinations are only administered 
To students who have completed the coursework and are deemed to be 
Ready to take the examination. 

For these reasons, this study examined two major sources of data independently 
and then attempted to draw conclusions across the two. The April 2000 CEE results were 
compared to fourth grade and eighth grade ELA and Math results for the year 2000, 
under the caveat that these would be district level analyses. District level Regents results 
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taken from a sample of schools drawn from the cohort surveys were matched with district 
level results on the fourth and eighth grade examinations, as well. 



Hypotheses 

Based on the design of the tests, we expect the following in support of the 
construct validity of ELA-8: 

1 . The best predictor of the probability of passing the Regents 
CEE is achieving level 2 or higher on the ELA-8. That is, 
districts with disappropriate representation of 

students in level 1 should have the lowest CEE 
passing rates (higher positive correlations for being in 
level 2 or higher or negative correlations for being 
in level 1); 

2. The prediction of passing the Regents CEE, based 
on performance on ELA-8 should be at or above 

a correlation of .35 (.1225 r-square). 



Methods 



Overview of Methodology 

In view of all these prescriptions, this study used the two data sources cited above. 
It should be kept in mind that longitudinal student level information may not be the most 
sensitive predictive source, for reasons given earlier, and that the level of scoring of a 
district across tests is a valuable indicator of an instructional program that should affect 
student performance across grades. Therefore, test sensitivity to this cumulative 
effectiveness of the standards-based academic program should, as well, be a good source 
of predictive evidence. 

Analyses 

Zero-order (bivariate) and multivariate correlations were computed among the 
spring 2000 results from the three instruments (ELA-4, ELA-8, and the CEE) and 
subsections of the grade four and grade eight examinations using multivariate general 
linear models and stepwise techniques. The subsections included the open-ended and 
multiple choice totals taken separately and the Standards Performance Indicators (SPIs), 
which are scores based performance on the three Learning Standards assessed on the 
ELA examinations. 

The multivariate stepwise regressions utilized a contribution of .05 to the total r- 
square as a criterion for retaining the independent variables. The dependent variables for 
these analyses were the district mean score on the April 2000 CEE and the probabilities 
of achieving a 55, 65, or 85 on the CEE from both data sets, as transformed into the linear 
delta scale. Dependent variables included the components (SPI's, multiple choice totals 
and open-ended totals) of ELA-4 and ELA-8 to predict the April 2000 mean score, and 



the probabilities of achieving a Level 1, Level 2 and above, Level 3 and above, or Level 4 
on ELA-4 and ELA-8 transformed into delta values for both data sets. It was reasoned 
that the contribution of the ELA-4 achievement levels would be the greater contributors 
to achieving the lower levels of ELA-8 and that achieving the three levels of the CEE 
would be the greater contributors to predictions of achieving the higher levels of ELA-8. 

The scale scores on the ELA-4 range from 455 to 800, and the scale scores on the 
ELA-8 range from 517 to 830. 

The performance levels for the grade four and grade eight examinations each 
range from 1, in definite need of academic intervention, to 4, advanced achievement. 
Levels 1 and 2 are indications of insufficient achievement of the Learning Standards. 

The three cutoff scores demarcating the four levels of ELA-4 are 601, 645, and 692. The 
cutoff scores demarcating the four levels of ELA-8 are 662, 701, and 739. 

Discriminant analyses were employed to determine if the probabilities of 
achieving these four levels on ELA-4 and ELA-8, as described above, could be used to 
classify performance into categories of performance on the CEE based on (proportions 
below 65 and 65 and above). A priori and posterior probabilities of achieving a 65 or 
higher were computed for each delta value related to probabilities of achieving Level 2 
and above and Level 3 and above on ELA-4 and on ELA-8. If the discriminant analyses 
classifications were reliable, then the distributions on the ELA-4 and the ELA-8 at which 
the probabilities for each CEE category were highest could the utility of those scoring 
levels in terms of future success on the CEE. Again, while these data do not track 
individual students, they do gain the advantage of freedom from contamination from 
intervening variables and they directly address the accountability function of the ELA-4 
and the ELA-8, as described above. 
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Samples 



April 2000 data. The April 2000 CEE was administered to 6,522 students. They 
achieved a mean scale score of 55.83 and a standard deviation of 12.99 (see Table 1). 

The mean of the district means (sampling distribution) was also 55.83, indicating good 
representation of lower and higher scoring districts by size. The standard deviation of 
district means was 7.12. The a priori probability of achieving a 65 or higher was 0.24, 
with a standard deviation of .43. The correlation between district mean scores and the 
student scores was 0.548. 

June 2000 district level data. Three hundred and eight school districts with 
students who took the June 2000 CEE were sampled for the study. Of these, 306 had 
complete data on the three tests. The means and standard deviations on the CEE were not 
available (although, with some distributional assumptions, means and standard deviations 
could be estimated), but the percentages of children scoring 0-54, 55-64, 65-84, and 85 
and above were taken from the statewide survey results. The average percentage at 65 or 
higher was 79.85, a much higher skilled population, as expected, than the April 2000 
group. 

These 306 districts were sampled because of their location in counties that 
represent major demographic attributes of New York State. These counties included 
New York City (counted as a single county); the counties of each of the Big Four cities 
(Erie, Monroe, Onondagua, and Westchester); the suburban counties of Rockland, 

Orange, Nassau, Suffolk, Dutchess, and Putnam; Albany, Rensselaer, and Schenectady 
because of their diversity in community types and upstate locations; Chautauqua, 
Chemung, Niagara, and Ulster for their inclusion of both urban and rural communities; 
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and Broome, Chenango, Fulton, and Genesee for their upstate representation of small 
communities. 

Scales for Analyses 

District level percentages or proportions achieving each of the four levels on the 
grade eight and grade four examinations and on the CEE were all converted to an interval 
scale (the delta scale described earlier, which is generally used to estimate test item 
difficulty) in order to perform the mathematical manipulations needed for this study. 

This scale has a mean of 13 and a standard deviation of -4, so that the smaller percentage 
achieving any performance level, the higher the delta, which can be interpreted as the 
higher the difficulty of achieving that level. Means and standard deviations of district 
level data are presented in Table 2. 



Table 1 



Means, Standard Deviations, and Correlations 



among the 2000 ELA-4, ELA-8, 



and the April 2000 



Special Administration of the CEE 



Correlations with CEE 



Test Variable 


Mean S.D. 


Pf85+) Pf65+) Pf55+) 


Probabilities 


ELA-4 Scale 


655.09 15.27 


0.254 


0.393 0.441 




m.c. 


20.28 1.46 


0.221 


0.393 0.451 




o.e. 


8.85 1.03 


0.251 


0.345 0.379 




p(level I)'* 


19.69 2.45 


-0.262 


-0.431 -0.483 


0.047 


p^evel 2+) 


6.31 2.45 . 


0.262 


0.431 0.483 


0.953 


p^evel 3+) 


11.77 1.91 


0.251 


0.386 0.435 


0.619 


p(level 4) 


17.41 1.81 


0.257 


0.345 0.370 


0.136 



Scale 




697.25 


11.29 


0.337 


0.484 


0.522 




m.c. 




19.72 


1.33 


0.297 


0.490 


0.535 




o.e. 




10.87 


1.11 


0.334 


0.454 


0.492 




p(level 


1) 


18.25 


1.81 


-0.320 


-0.461 -0.519 


0.100 


p(level 


2+) 


7.75 


1.81 


0.319 


0.460 


0.518 


0.905 


p(level 


3+) 


13.62 


1.64 


0.301 


0.475 


0.511 


0.438 


p(level 


4) 


18.69 


1.55 


0.356 


0.379 


0.380 


0.078 




11 18 



Table 2 



Means, Standard Deviations, and Correlations 



Among Grade ELA-4, ELA-8, 



and the CEE, Spring 2000, for 306 School Districts 









Correlations with CEE 




Test Variable 


Mean 


S.D. 


P('85+'l ?( 65 +) 


P155+1 


Probabilities 


ELA-4 Scale 


665.18 


14.94 


0.583 


0.484 


0.312 




m.c. 


21.22 


1.28 


0.560 


0.504 


0.312 




o.e. 


9.45 


1.02 


0.558 


0.448 


0.284 




p(level 1)^ 


21.32 


2.41 


-0.412 


-0.309 


-0.147 


0.015 


p(level 2+) 


4.68 


2.41 


0.412 


0.309 


0.147 


0.981 


p(level 3+) 


10.50 


2.33 


0.567 


0.520 


0.316 


0.735 


p(level 4) 


16.46 


1.92 . 


0.585 


0.501 


0.366 


0.194 



ELA-8 Scale 


704.37 


16.34 


0.502 


0.429 


0.330 




m.c. 


20.52 


1.58 


0.506 


0.448 


0.346 




o.e. 


11.54 


1.64 


0.528 


0.468 


0.359 




p(level 1) 


19.63 


2.63 


-0.381 


-0.343 


-0.243 


0.048 


p(level 2 +) 


6.37 


2.,63 


0.382 


0.344 


0.244 


0.952 


p(level 3-I-) 


12.60 


2.48 


0.572 


0.51.4 


0.385 


0.540 


p(level 4) 


18.04 


2.06 


0.600 


0.515 


0.395 


0.104 
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Specific Procedures 



Two sets of stepwise analyses were used to determine the contribution of multiple 
grade 8 and grade 4 ELA variables. The two sets of analyses described earlier differed in 
terms of the choice of dependent variables, one focusing on within-examinee variables, 
and the other on between examinee variables. Because the within-examinee variables 
from the ELA-4 and ELA-8 are secondary to the instructional placement of children, and 
to simplify the considerable volume of results, stepwise analyses of these variables were 
limited to predicting the scale score mean on the CEE. These variables included: cluster 
(open-ended) totals (clus4 and clus8, respectively), the multiple choice totals (mc4 and 
mc8, respectively), and the three standards performance indices from each examination 
(spi41-spi43 and spi81-spi83, respectively). 

The second set of stepwise analyses assessed the contribution of the 
representation on the ELA-4 and ELA-8 in each of the four scoring levels for each test to 
the performance of students from those districts on the April CEE. To assess this, the 
proportions of students scoring in level 1, in levels 2 - 4, in level 3-4, and in level 4, 
just as depicted in Tables 1 and 2 above, were calculated and then transformed to the 
linear delta scale. The dependent variables for this set of analyses included: Proportion 
of students in the districts represented on the April CEE achieving 55 or higher, 
achieving 65 or higher, and achieving 85 or higher; and average scale score of 
participating districts. 

Two discriminant analyses were employed to classify school districts into one of 
five groups related to proportion of students achieving scores of 65 or higher on the CEE. 
These groups were: 0 - .20 achieving 65 or higher, .21 - .40 achieving 65 or higher, .41 - 
.60 achieving 65 or higher, .61 to .80 achieving 65 or higher, and .81 - 1.00 achieving 65 
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or higher. The prior probabilities were set at the proportional representation of these 
classifications over all school districts participating in the April administration of CEE. 
The first analysis found the average district scale scores on the ELA-4 demarcating group 
membership in each of these five classifications, and the second analysis found that score 
on the ELA-8. 



306 Sampled Districts 

The analyses conducted on the sample of 306 school districts were the same as 
those described above for the April CEE administration, with one exception. Because no 
scale score average was available for the school districts on the CEE, none of the 
analyses employed this as a dependent variable. The proportional representation of the 
school districts at CEE scores of 0 - 54, 55 - 64, 65 - 84, and 85 - 100 remained as 
dependent variables. 
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Results 



General Findings 

Partial correlations and standard errors of prediction for the April 2000 stepwise 
analyses are given in Table 3. Table 4 shows the results of the discriminant analyses for 
that sample. Tables 5 and 6, respectively, give the outcomes of these analyses for the 306 
districts sampled. 



Table 3 



Stepwise Regression Results Using 
Components of the ELA-4 and ELA-8 and 
Proportions Achieving Levels 1 - 4 on the ELA-4 
and ELA-8 to Predict Performance on 
the April 2000 Administration of The 
Regents Comprehensive Examination in English (CEE) 



Independent 


Dependent 


Partial 


Model 


Parameter 


Standard 


Variable 


Variable 


R-Sauare 


R-Sauare 


Estimate 


Error 


Sp81 


Scale mean 


0.380 


0.380 


2.621 


0.195 


Sp82 


Scale mean 


0.016 


0.396 


-1.911 


0.208 


Clus4 


Scale mean 


0.001 


0.397 


1.944 


0.198 


Sp43 


Scale mean 


0.009 


0.406 


-0.281 


0.032 


MC4 


Scale mean 


0.008 


0.410 


1.160 


1.169 


Sp42 


Scale mean 


0.005 


0.415 


-0.231 


0.044 


Sp41 


Scale mean 


0.001 


0.416 


0.078 


0.036 


Intercept = -1.394 










P(level 3+, 8)^ 


’ Scale mean 


0.346 


0.346 


-1.667 


0.219 


P(level 2+, 8) 


Scale mean 


0.016 


0.362 


-2.271 


0.681 


P(level 4, 4) 


Scale meaii 


0.002 


0.364 


0.080 


0.031 


P(level 1, 4) 


Scale mean 


0.003 


0.367 


0.103 


0.026 


P(level 4, 8) 


Scale mean 


0.002 


0.370 


0.244 


0.151 


P(level 1, 8) 


Scale mean 


0.000 


0.370 


0.919 


0.662 


P(level 2 +, 4) 


Scale mean 


0.000 


0.370 


5.991 


7.862 


Intercept = -47.300 










P(Ievel 3+, 8) 


P(55+) 


0.255 


0.255 


1.253 


0.152 


P(level 1, 8) 


P(55+) 


0.012 


0.267 


-0.579 


0.102 


P(level 1, 4) 


P(55+) 


0.004 


0.271 


-0.078 


0.015 


P(level 3+, 4) 


P(55+) 


0.003 


0.275 


-0.056 


0.016 


P(level 4, 8) 


P(55+) 


0.002 


0.277 


-0.315 


0.100 



Intercept = 10.676 




16 



23 



Table 3 



Stepwise Regression Results Using 
Components of the ELA-4 and ELA-8 and 
Proportions Achieving Levels 1 - 4 on the ELA-4 
and ELA-8 to Predict Performance on 
the April 2000 Administration of The 
Regents Comprehensive Examination in English (CEE) 



Independent 


Dependent 


Partial 


Variable 


Variable 


R-Sauare 


P(level 3+, 8) 


P(65+) 


0.174 


P(level 4, 4) 


P(65+) 


0.002 


P(level 1, 8) 


P(65+) 


0.003 


P(level 1,4) 


P(65+) 


0.000 


P(level3+,4) P(65+) 


0.001 


P(level2+, 4) P(65+) 


0.000 


P(level 2+, 8) P(65+) 
Intercept = 151.707 


0.000 


P(level 4, 8) 


P(85+) 


0.113 


P(level3+,4) P(85+) 


0.010 


P(level 2+, 8) P(85+) 


0.006 


P(level 1, 8) 


P(85+) 


0.018 


P(level 3+, 8) P(85+) 


0.014 


P(level 4, 4) 


P(85+) 


0.000 


P^evel 1, 4) 


P(85+) 


0.000 


Intercept = -49.515 





Model 


Parameter 


Standard 


R-Sauare 


Estimate 


Error 


0.174 


0.978 


0.124 


0.176 


-0.043 


0.023 


0.179 


0.077 


0.450 


0.179 


-5.912 


5.793 


0.180 


-0.047 


0.019 


0.181 


-5.872 


5.793 


0.181 


0.409 


0.475 



0.113 


0.701 


0.074 


0.123 


-0.043 


0.013 


0.128 


3.237 


0.333 


0.146 


2.571 


0.324 


0.160 


-0.779 


0.107 


0.161 


-0.021 


0.015 


0.161 


-0.009 


0.011 
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Table 4 



Classiflcatory Discriminant Analyses 
of Proportions within School Districts 
Achieving 65 or Better on the Special 
April Administration of the CEE, 
Based on ELA-4 and ELA-8 Scale Scores 



Classification 

Variable Prob. P(scoring 65+): 
ELA-4 lev. 2+ 0.89 (delta=8.08) 

0.97 (delta=5.48) 

1.00 (delta= 1.00) 

lev. 3+0.70 (delta=10.88) 

ELA-8 lev. 2+ 0.95 (delta=6.44) 
lev. 3+0.54 (delta=12.60) 



Posterior Probabilities of Membership 



0-.20 .21-.40.41-.60.61-.80 .81-1.00 



0.271 


0.495 


0.148 


0.008 


0.078 


0.208 


0.234 


0.286 


0.197 


0.075 


0.058 


0.243 


0.332 


0.000 


0.367 


0.209 


0.254 


0.258 


0.196 


0.083 


0.188 


0.270 


0.291 


0.149 


0.101 


0.186 


0.282 


0.290 


0.154 


0.089 




Table 5 



Stepwise Regression Results Using 
Components of the ELA-4 and ELA-8 and 
Proportions Achieving Each Level or Higher on the ELA-4 
and ELA-8 to Predict Performance on 
the June 2000 Administration of The 
Regents Comprehensive Examination in 
English (CEE) for 306 Districts 



Independent 


Dependent 


Partial 


Variable 


Variable 


R-Sauare 


P(level 4, 8) 


P(55+) 


0.188 


P(level 4, 4) 


P(55+) 


0.013 


P(level 2+,4) 


P(55+) 


0.025 


P(level 3+, 8) 


P(55+) 


0.008 


P(level 3+, 4) P(55+) 
Intercept = -19.734 


0.005 


P(level 4, 8) 


P(65+) 


0.302 


P(level3+, 4) P(65+) 


0.054 


P(level 2 +, 4) P(65+) 


0.025 


P(level 3+, 8) P(65+) 
Intercept = -10.996 


0.002 


P(level 4, 8) 


P(85+) 


0.378 


P(level 3+, 4) 


P(85+) 


0.047 


P(level 1, 8) 


P(85+) 


0.021 


P(level 4, 4) 


P(85+) 


0.010 


P(level 3+, 8) 


P(85+) 


0.002 


Intercept = -5. 


123 





Model 


Parameter 


Standard 


R-Sauare 


Estimate 


Error 


0.188 


1.041 


0.349 


0.201 


0.715 


0.410 


0.225 


-0.566 


0.214 


0.233 


-0.419 


0.337 


0.238 


0.315 


0.301 



0.302 


0.873 


0.274 


0.355 


0.872 


0.191 


0.380 


-0.451 


0.169 


0.382 


-0.159 


0.233 



0.378 


0.729 


0.170 


0.426 


0.178 


0.122 


0.447 


0.178 


0.086 


0.456 


0.388 


0.200 


0.459 


-0.158 


0.184 
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Table 6 



Classificatory Discriminant Analyses 
of Proportions within School Districts 
Achieving 65 or Better on the June 
Regents Comprehensive Examination 
In English (CEE), 

Based on ELA-4 and ELA-8 Scale Scores 
For 306 Districts 



Classification 

Variable Prob. P(scoring 65+): 

ELA-4 lev. 2+ 0.81 (delta-9.48) 

0.89 (delta-8.08) 

0.95 (delta-6.44) 

.lev. 3+ 0.32 (delta-14.88) 

0.48 (delta-13.20) 

0.65 (delta-11.44) 

ELA-8 lev. 2+0.94 (delta-6.80) 

lev. 3+ 0.1 9 (delta-16.52) 

0.34 (delta-14.64) 

0.50 (delta-13.00) 



Posterior Probabilities of Membership 
0-.20 .21-.40.41-.60.61-.80 ,81-1.00 



0.186 


0.072 


0.364 


0.257 


0.122 


0.085 


0.024 


0.319 


0.344 


0.228 


0.049 


0.009 


0.117 


0.393 


0.432 


0.367 


0.075 


0.406 


0.115 


0.036 


0.082 


0.014 


0.349 


0.386 


0.170 


0.027 


0.004 


0.073 


0.445 


0.451 


0.029 


0.003 


0.040 


0.454 


0.477 


0.218 


0.059 


0.390 


0.037 


0.296 


0.041 


0.009 


0.199 


0.382 


0.369 


0.019 


0.004 


0.036 


0.459 


0.483 
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Conclusion 



Overview of Results 

The limitations discussed above on the possible interpretations of the data are 
central to the considerations of the evidence and also of the utility of predictive sources 
of evidence in support of a state assessment system in the environment of Learning 
Standards. This is particularly true of the April 2000 administration of the CEE, which 
was to a population that had continued to fail the test through multiple attempts. 
Nevertheless, the strength of relationships attests to the utilityof both the ELA-4 and the 
ELA-8 as predictors of performance on the Regents examination. Moreover, the very 
close association of certain ranges of the ELA-4 and ELA-8 scale scores with subsequent 
passing and failing the Regents is good support for the construct validity of the standard 
setting process. 



Hypotheses 

Achieving Level 3 on ELA-8 is consistently the best predictor in the April data 
for both passing the CEE at the 55 level and for passing the CEE at the 65 level. As the 
CEE passing criterion is raised to 65 and to 85, being in Level 3 becomes the more 
powerful predictor. Note that none of the grade four criteria are nearly as powerful as the 
grade eight performance variables. Regarding the data from the 306 districts, having 
more students in Level 4 on the grade eight ELA is an obviously strong predictor of 
performance on the CEE. 

The discriminant analyses show that a high proportion of being at or above Level 
2 on both the ELA-4 and the ELA-8 was related to having at least .40 to .60 of the 
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students in the district pass the Regents CEE at the 65 level. Moreover, the analyses 
show clearly that having at least half of the students achieve Level 3 or higher on both the 
ELA-4 and the ELA-8 discriminates districts that have at least .40 to .60 pass the CEE 
Regents at the 65 level. 

Predictive Evidence of Validity 

The issue of predictive evidence, in its own, needs to be carefully considered. It 
is not clear that a study relating the scores of the same individuals over years in which 
there has been systematic exposure to intervening variables is necessarily the best source 
of this evidence. It may well be that, with some refinement, these district level data, in 
conjunction with individual data, and perhaps with imbedded items across tests to permit 
equating, could continue to systematically used to test validity hypotheses. Nevertheless, 
we must interpret such evidence with caution, never forgetting that the function of these 
state assessments is to certify the achievement of the Learning Standards. 

Evaluating the sources of predictive evidence becomes an important process when 
the purposes of the tests are so specific. Showing that the percentage of children at or 
above Level 3 on the ELA-8, for example, while on the surface seems like a good 
criterion for predictive evidence, may not be as strong as it seems. For example, in 
defining the four levels of performance on the grade eight and grade four examinations, 
the standard setting studies.referred to the extent of acquisition of the appropriate 
Learning Standards for the grade. Reference was not made to whether Level 1 or Level 2 
should correspond to a particular level of passing the Regents. Rather, each successive 
level of achievement should be monotonically related to the probability of ultimately 
passing the Regents on the basis of achieving the Learning Standards. 



Therefore, for the purposes of predictive evidence, Level 1, according to 
definition, indicates that the appropriate standards have not been achieved, and therefore 
identification in at least level 1 should be a powerful predictor of passing the CEE at 55, 
65, or 85. Level 2, which indicates at least partial deficiency should also be related to 
lower passing rates on the Regents at any score, while higher probabilities of achieving 
Levels 3 and 4 should evidence positive relationships. For the purposes of academic 
intervention, then, identification in Level 2 should trigger intervention, because the 
Standards have not been fully achieved. 

Simply put, there is no reason to expect that proportions of children performing at 
one level of a lower grade test should match proportions performing at a level on a 
subsequent State examination. Rather, we would expect that the probability of passing 
the Regents at any cut score increases as the probability of performance at each level of 
the grade eight test increases. That is why the multivariable correlative analyses are such 
a central source of predictive evidence. 

The weight of the evidence is overwhelming that this relationship is precisely 
what is observed. To examine its nature, one final analysis was undertaken. Using data 
from the 306 districts, it was hypothesized that as the scale scores on the ELA-8 
increased, that the differences between the probability of scoring at or above that scale 
score and the probabilities of achieving certain CEE scores (55, 65, and 85) would first 
decrease and then increase. That is, as the ELA-8 criteria for passing increase, the 
probabilities associated with those scores would decrease until they virtually matched the 
probabilities of passing the CEE. Then, as the criterion scores continue to increase, the 
passing rates on the ELA-8 would drop and the probabilities for passing the CEE would 
be higher. More importantly, however, is that the higher the CEE score used as a 
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criterion (55, 65, or 85), the higher the ELA-8 score with the most similar proportion 
achieving that score (the lower the difference between the proportions achieving scores 
on each test). 

Table 7 presents the differences in delta units between the proportion at or above 
each ELA-8 score and proportions achieving 55, 65, and 85 on the CEE. These 
proportion differences are squared, averaged within districts and the square root is taken 
of that average to approximate errors of prediction of one test onto the other based on 
rates achieving each score (note, the Appendix provides the same statistic for the reader’s 
information but taken as proportions rather than transformed into the delta metric). It is 
clear that the pattern of these decreases in the errors supports the hypotheses. It is also 
clear, from the bold entries (the lowest error), that the agreement in proportions achieving 
65 on the CEE and the ELA-8 are best at the ELA-8 scale score of 677, somewhat about 
the Level 2 cut score of 662. Thereafter, the agreement with achieving a 65 on the CEE 
remains better than of achieving a 55 on the CEE for each score of the ELA-8. Similarly, 
the proportions scoring at or above 85 on the CEE are most similar to the proportions 
scoring at or above 73 Ion the ELA-8. Again, the magnitudes of these agreements are not 
in themselves the test of predictive evidence. However, the mono tonicity of the 
relationships clearly lends support to the predictive evidence. In a sense, these analyses 
address the issues of what would the cut scores have to be to maximize the agreement in 
proportions achieving that score and proportions passing the Regents examination. The 
answer, taking into account the statistical error inherent in these sampling procedures, is 
clearly right around where the cut scores are now! Figure 1 illustrates the relationship 
between achieving each scale score on ELA-8 and achieving a 55, 65, or 85 on the CEE. 
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Table 8 presents a representation of the relationship between an independent 
variable of each ELA-8 scale score and the dependent variable of the differences in delta 
units between the proportions at or above 55, 65, and 85 on the Regents CEE and at or 
above each of those scale scores. The relationship is remarkably quadratic, as predicted: 
as the criterion ELA-8 score increases, the differences in proportions passing ELA-8 and 
passing the Regents CEE decreases and then increases, as you would expect with a 
monotonic relationship between the passing rates on the two examinations. The 
quadratic relationship of achieving a 65 on the CEE is; .00167 (criterion ELA-8 score 
squared) - 2.2489 (criterion ELA-8 score) +.759.2543. When we differentiate and solve 
for the minima, we attain a score of 673.32, between the Level 2 and Level 3 cutoffs. 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 








Scale 


Agreement ErrOi 


■ of Probability of Achieving 


Score 






85 


610 


17.64 


15.08 


8.20 


611 


3.68 


6.24 


13.12 


612 


3.68 


6.24 


13.12 


613 


3.68 


6.24 


13.12 


614 


3.68 


6.24 


13.12 


615 


3.68 


6.24 


13.12 


616 


3.68 


6.24 


13.12 


617 


3.68 


6.24 


13.12 


618 


2.56 


5.12 


12.00 


619 


2.56 


5.12 


12.00 


625 


2.56 


5.12 


12.00 


626 


2.56 


5.12 


12.00 


627 


2.56 


5.12 


12.00 


628 


2.56 


5.12 


12.00 


629 


2.56 


5.12 


12.00 


630 


2.56 


5.12 


12.00 


631 


2.56 


5.12 


12.00 


632 


1.88 


4.44 


11.32 


633 


1.88 


4.44 


11.32 


634 


1.88 


4.44 


11.32 


635 


1.88 


4.44 


11.32 


636 


1.88 


4.44 


11.32 


637 


1.88 


4.44 


11.32 


638 


1.88 


4.44 


11.32 


639 


6.36 


8.92 


15.80 



CEE Scores at or above 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 

Scale Agreement Error of Probability of Achieving CEE Scores at or above 



Score 




65 


85 


640 


1.36 


, 3.92 


10.80 


641 


1.36 


3.92 


10.80 


642 


1.36 


3.92 


10.80 


643 


1.36 


3.92 


10.80 


644 


0.92 


3.48 


10.36 


645 


0.92 


3.48 


10.36 


646 


0.92 


3.48 


10.36 


647 


0.92 


3.48 


10.36 


648 


0.56 


3.12 


10.00 


649 


0.56 


3.12 


10.00 


650 


0.56 


3.12 


10.00 


651 


0.56 


3.12 


10.00 


652 


0.24 


2.80 


9.68 


653 


0.24 


2.80 


9.68 


654 


0.24 


2.80 


9.68 


655 


0.00^ 


2.56 


9.44 


656 


0.00 


2.56 


9.44 


657 


0.28 


2.28 


9.16 


658 


0.28 


2.28 


9.16 


659 


0.52 


2.04 


8.92 


660 


0.52 


2.04 


8.92 


661 


0.72 


1.84 


8.72 


662 


0.72 


1.84 


8.72 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


65 


S5 


663 


0.96 


1.60 


8.48 


664 


0.96 


1.60 


8.48 


665 


1.12 


1.44 


8.32 


666 


1.12 


1.44 


8.32 


667 


1.32 


1.24 


8.12 


668 


1.48 


1.08 


7.96 


669 


1.48 • 


1.08 


7.96 


670 


1.68 


0.88 


7.76 


671 


1.84 


0.72 


7.60 


672 


1.96 


0.60 


7.48 


673 


1.96 


0.60 


7.48 


674 


2.12 


0.44 


7.32 


675 


2.28 


0.28 


7.16 


676 


2.40 


0.16 


7.04 


677 


2.56 


0.00 


6.88 


678 


2.56 


0.00 


6.88 


679 


2.68 


0.12 


676 


680 


2.80 


0.24 


6.64 


Os 

00 

oc 


2.96 


0.40 


6.48 


682 


3.08 


0.52 


6.38 


683 


3.20 


0.64 


6.24 


684 


3.32 


0.76 


6.12 


685 


3.44 


0.88 


6.00 


686 


3.64 


1.08 


5.80 


687 


3.76 


1.20 


5.68 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 








688 


3.88 


1.32 


5.56 


689 


4,00 


1.44 


5.44 



690 


4.08 


1.52 


5.36 


691 


4.20 


1.64 


5.24 


692 


4.40 


1.84 


5.04 


693 


4.52 


1.96 


4.92 


694 


4.64 


2.08 


4.80 


695 


4.72 


2.16 


4.72 


696 


4.92 


2.36 


4.52 


697 


5.04 


2.48 


4.40 


698 


5.12 


2.56 


4.32 


699 


5.32 


2.76 


4.12 


700 


5.44 


2.88 


4.00 


701 


5.48 


2.92 


3.96 


702 


5.80 


3.24 


3.64 


703 


5.84 


3.28 


3.60 


704 


5.96 


3.40 


3.48 


705 


6.04 


3.48 


3.40 


706 


6.24 


3.68 


3.20 


707 


6.36 


3.80 


3.08 


708 


6.44 


3.88 


3.00 


709 


6.64 


4.08 


2.80 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 






85 


710 


6,76 


4.20 


2.68 


711 


6.88 


. 4.32 


2.56 


712 


6.96 


4.40 


2.48 


713 


7.20 


4.64 


2.24 


714 


7.28 


4.72 


2.16 


715 


7.40 


4.84 


2.04 


716 


7.52 


4.96 


1.92 


717 


7.64 


5.08 


1.80 


718 


7.84 


5.28 


1.60 


719 


7.96 


5.40 


1.48 


720 


8.08 


5.52 


1.36 


721 


8.20 


5.64 


1.24 


111 


8.32 


5.76 


1.12 


111 


8.48 


5.92 


0.96 


724 


8.60 


6.04 


0.84 


725 


8.72 


6.16 


0.72 


726 


8.72 


6.16 


0.72 


111 


8.88 


6.32 


0.56 


728 


9.00 


6.44 


0.44 


729 


9.16 


6.60 


0.28 


730 


9.32 


6.76 


0.12 


731 


9.44 


6.88 


0.00 


111 


9.60 


7.04 


0.16 


111 


9.60 


7.04 


0.16 


734 


9.80 


7.24 


0.36 
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Table 7 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Delta Units) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


65 




735 


9.96 


. 7.40 


0.52 


736 


10.16 


7.60 


0.72 


737 


10.16 


7.60 


0.72 


738 


10.32 


7.76 


0.88 


739 


10.32 


7.76 


0.88 


740 . 


10.56 


8.00 


1.12 




Table 8 



Multivariate Regression of Agreement Error in Delta Units 
between Proportions Achieving at or above Each 
Scale Score on the ELA-8 and Proportions Achieving 
55, 65, and 85 on the Regents CEE onto 
the ELA-8 Scale Scores 



CEE 

Passing 

Score 


Independent 

Variable 


Partial 

R-Sauare 


Model 

R-Square 


Parameter 

Estimate 


Standard 

Error 


55 


ELA-8 Score 


0.052 


0.052 


-0.875 


0.103 




ELA-8 Score 
squared 


0.855 


0.907 


0.000 


0.000 


Intercept = 


269.778 










65 


ELA-8 Score 


0.498 


0.498 


-1.947 . 


0.112 




ELA-8 Score 
squared 


0.292 


0.790 


0.001 


0.000 


Intercept = 


646.743 










85 


ELA-8 Score 


0.946 


0.946 


-0.457 


0.074 




ELA-8 Score 


0.009 


0.955 


0.000 


0.000 



squared 

Intercept =191.594 




32 



39 



Figure 1 



Plot of Agreement Error 
of Proportions Achieving 
at or Above ELA-8 Scale Scores and 
Regents CEE Scale Scores of 55 r^l'^) , 
65 ("h"), and 85 ("d") , 306 Districts 
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Appendix 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


65 


85 


610 


0.92 


.0.78 


0.17 


611 


0.07 


0.21 


0.82 


612 


0.07 


0.21 


0.82 


613 


0.07 


0.21 


0.82 


614 


0.07 . 


0.21 


0.82 


615 


0.07 


0.21 


0.82 


616 


0.07 


0.21 


0.82 


617 


0.07 


0.21 


0.82 


618 


0.06 


0.20 


0.81 


619 


0.06 


0.20 


0.81 


620 


0.06 


0.20 


0.81 


621 


0.06 


0.20 


0.81 


622 


0.06 


0.20 


0.81 


623 


0.06 


0.20 


0.81 


624 


0.06 


0.20 


0.81 


625 


0.06 


0.20 


0.81 


626 


0.06 


0.20 


0.81 


627 


0.06 


0.20 


0.81 


628 


0.06 


0.20 


0.81 


629 


0.06 


0.20 


0.81 


630 


0.06 


0.20 


0.81 


631 


0.06 


0.20 


0.81 


632 


0.05 


0.19 


0.80 


633 


0.05 


0.19 


0.80 


634 


0.05 


0.19 


0.80 



O 




34 



Appendix 

Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 . 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 






S5 


635 


0.05 


0.19 


0.80 


636 


0.05 


0.19 


0.80 


637 


0.05 


0.19 


0.80 


638 


0.05 


0.19 


0.80 


639 


0.08 


0.22 


0.83 


640 


0.04 


0.18 


0.79 


641 


0.04 


0.18 


0.79 


642 


0.04 


0.18 


0.79 


643 


0.04 


0.18 


0.79 


644 


0.03 


0.17 


0.78 


645 


0.03 


0.17 


0.78 


646 


0.03 


0.17 


0.78 


647 


0.03 


0.17 


0.78 


648 


0.02 


0.16 


0.77 


649 


0.02 


0.16 


0.77 


650 


0.02 


0.16 


0.77 


651 


0.02 


0.16 


0.77 


652 


0.01 


0.15 


0.76 


653 


0.01 


0.15 


0.76 


654 


0.01 


0.15 


0.76 


655 


0.00 


0.14 


0.75 


656 


0.00 


0.14 


0.75 


657 


0.01 


0.13 


0.74 


658 


0.01 


0.13 


0.74 


659 


0.02 


0.12 


0.73 
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Appendix 



Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


65 


M 


660 


0.02 


0.12 


0.73 


661 


0.03 


0.11 


0.72 


662 


0.03 


0.11 


0.72 


663 


0.04 


0.10 


0.71 


664 


0.04 


0.10 


0.71 


665 


0.05 


0.09 


0.70 


666 


0.05 


0.09 


0.70 


667 


0.06 


0.08 


0.69 


668 


0.07 


0.07 


0.68 


669 


0.07 


0.07 


0.68 


670 


0.08 


0.06 


0.67 


671 


0.09 


0.05 


0.66 


672 


0.10 


0.04 


0.65 


673 


0.10 


0.04 


0.65 


674 


0.11 


0.03 


0.64 


675 


0.12 


0.02 


0.63 


676 


0.13 


0.01 


0.62 


677 


0.14 


0.00 


0.61 


678 


0.14 


0.00 


0.61 


679 


0.15 


0.01 


0.60 


680 


0.16 


0.02 


0.59 


681 


0.17 


0.03 


0.58 


682 


0.18 


0.04 


0.57 


683 


0.19 


0.05 


0.56 


684 


0.20 


0.06 


0.55 
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Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


§5 












685 


0.21 


0.07 


0.58 


686 


0.23 


0.09 


0.57 


687 


0.24 


0.10 


0.56 


688 


0.25 


0.11 


0.55 


689 


0.26 


0.12 


0.53 


690 


0.27 


0.13 


0.48 


691 


0.28 


0.14 


0.47 


692 


0.30 


0.16 


0.45 


693 


0.31 


0.17 


0.44 


694 


0.32 


0.18 


0.43 


695 


0.33 


0.19 


0.42 


696 


0.35 


0.21 


0.40 


697 


0.36 


0.22 


0.39 


698 


0.37 


0.23 


0.38 


699 


0.39 


0.25 


0.36 


700 


0.40 


0.26 


0.42 


701 


0.41 


0.27 


0.40 


702 


0.43 


0.29 


0.39 


703 


0.44 


0.30 


0.38 


704 


0.45 


0.30 


0.36 


705 


0.46 


0.32 


0.29 


706 


0.48 


0.34 


0.27 


707 


0.49 


0.35 


0.26 


708 


0.50 


0.36 


0.25 


709 


0.52 


0.38 

37 


0.23 

44 
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Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 




85 


710 


0.53 


0.39 


0.22 


711 


0.54 


0.40 


0.21 


712 


0.55 


0.41 


0.20 


713 


0.57 


0.43 


0.18 


714 


0.58 


0.44 


0.17 


715 


0.59 


0.45 


0.16 


716 


0.60 


0.46 


0.15 


717 


0.61 


0.47 


0.14 


718 


0.63 


0.49 


0.12 


719 


0.64 


0.50 


0.11 


720 . 


0.65 


0.51 


0.10 


721 


0.66 


0.52 


0.09 


111 


0.67 


0.53 


0.08 


111 


0.68 


0.54 


0.07 


724 


0.69 


0.55 


0.06 


725 


0.70 


0.56 


0.05 


726 


0.70 


0.56 


0.05 


111 


0.71 


0.57 


0.04 


728 


0.72 


0.58 


0.03 


729 


0.73 


0.59 


0.02 
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Agreement Error Conditioned on ELA-8 Scale Score, 
Probability of Students at Each Scale Score 
Passing the CEE at 55, 65, and 85, 

Based on 306 Districts (Proportions, Absolute Differences) 



ELA-8 

Scale Agreement Error for Probability of Achieving CEE Scores at or above 



Score 


55 


65 


M 


730 


0.74 


0.60 


0.01 


731 


0.75 


0.61 


0.00 


732 


0.76 


0.62 


0.01 


733 


0.76 


0.62 


0.01 


734 


0.77 


0.63 


0.02 


735 


0.70 


0.64 


0.03 


736 


0.70 


0.65 


0.04 


737 


0.71 


0.65 


0.04 


738 


0.82 


0.66 


0.05 


739 


0.83 


0.66 


0.05 


740 


0.81 


0.67 


0.06 
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* American Educational research Association, American Psychological Association, and National Council 
on Measurement in Education Standards for Educational and Psychological 
Testing (Author, Washington, DC: 1999), see especially p. 17. 

‘^Ibid., p. 13. 

^ Ibid.y p. 14. 

^ Means, standard deviations, and correlations are in delta units. Because delta 
units are a difficulty scale, the correlations have been transformed from 
negative values to positive values. 

^Means, standard deviations, and correlations are in delta units. Because delta 
units are a difficulty scale, the correlations have been transformed from 
negative values to positive values. 

^ Should be read as the probability of achieving level 1 on the grade 8 ELA. 



g 

Note italicized ELA-8 scores indicate the first score of a level of 
performance, e.g., the lowest level 2 score. 
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